The performance analysis of linux networking - Packet receiving

The computing models for HEP experiments are becoming ever more globally distributed and grid-based, both for technical reasons (e.g., to place computational and data resources near each other and the demand) and for strategic reasons (e.g., to leverage technology investments). To support such computing models, the network and end systems (computing and storage) face unprecedented challenges. One of the biggest challenges is to transfer physics data sets – now in the multi-petabyte (10 bytes) range and expected to grow to exabytes within a decade – reliably and efficiently among facilities and computation centers scattered around the world. Both the network and end systems should be able to provide the capabilities to support high bandwidth, sustained, endto-end data transmission. Recent trends in technology are showing that although the raw transmission speeds used in networks are increasing rapidly, the rate of advancement of microprocessor technology has slowed down over the last couple of years. Therefore, network protocol-processing overheads have risen sharply in comparison with the time spent in packet transmission, resulting in the degraded throughput for networked applications. More and more, it is the network end system, instead of the network, that is responsible for degraded performance of network applications. In this paper, the Linux system’s packet receive process is studied from NIC to application. We develop a mathematical model to characterize the Linux packet receive process. Key factors that affect Linux systems’ network performance are analyzed. PACKET RECEIVING PROCESS Figure 1 demonstrates generally the trip of a packet from its ingress into a Linux end system to its final delivery to the application [1][2][3]. In general, the packet’s trip can be classified into three stages: • Packet is transferred from network interface card (NIC) to ring buffer. The NIC and device driver manage and controls this process. • Packet is transferred from ring buffer to a socket receive buffer, driven by a software interrupt request (softirq) [2][4]. The kernel protocol stack handles this stage. • Packet data is copied from the socket receive buffer to the application, which we will term the Data Receiving Process. In the following sections, we detail these three stages. It will be assumed that the medium is Ethernet-like and that the driver uses the Linux “New API” (NAPI) to reduce interrupt load on the CPU. NIC Hardware Network Application Traffic Sink Ring Buffer Socket RCV Buffer SoftIrq Process Scheduler TrafficSource IP Processing TCP/UDP Processing SOCK RCV SYS_CALL DMA Kernel Protocol Stack Data Receiving Process NIC & Device Driver Figure 1 Linux Networking Subsystem: Packet Receiving Process NIC and Device Driver Processing The Linux kernel uses a structure sk_buff [2] to hold any single packet up to the MTU (Maximum Transfer Unit) of the network. The device driver maintains a “ring” of these packet buffers, known as a “ring buffer,” for packet reception (and a separate ring for transmission). A ring buffer consists of a deviceand driver-dependent number of packet descriptors. To be able to receive a packet, a packet descriptor should be in “ready” state, which means it has been initialized and pre-allocated with an empty sk_buff memory-mapped into I/O space for DMA. When a packet comes, one of the ready packet descriptors in the reception ring will be used, the packet will be transferred into the pre-allocated sk_buff, and the descriptor will be marked as used. A used packet descriptor should be reinitialized and refilled with an empty sk_buff as soon as possible for further incoming packets. If a packet arrives and there is no ready packet descriptor in the reception ring, it will be discarded. Figure 2 shows a general packet receiving process at NIC and device driver level. When a packet is received, it is transferred into main memory and an interrupt is raised only after the packet is accessible to the kernel. When CPU responds to the interrupt, the driver’s interrupt handler is called, which schedules a softirq. It puts a reference to the device into the poll queue of the interrupted CPU and disables the NIC’s receive interrupt until the packets in its ring buffer are processed. * [email protected] † [email protected]